Codecov Report

❌ Patch coverage is 51.47059% with 66 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.17%. Comparing base (22d82c1) to head (6a96e4c).

@@            Coverage Diff             @@
##           master    #1043      +/-   ##
==========================================
- Coverage   81.46%   81.17%   -0.29%     
==========================================
  Files         101      101              
  Lines       13282    13411     +129     
==========================================
+ Hits        10820    10887      +67     
- Misses       2204     2254      +50     
- Partials      258      270      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

OK, I couldn't resist and tried this locally with BenchmarkSchedulingThroughPut().

There is a significant decrease in throughput, which baffled me a bit, but then I realized what was going on. This test is a bit synthetic, so not really realistic. The first node in the B-tree is always a "hit", so this piece of code slows things down, because it always has to construct at least a single batch (eg. 24), then walk through the nodes even if it's not needed. The first node of the batch is always the one that we need, so we perform unnecessary verification of predicates for the remaining 23 nodes, essentially wasting CPU cycles.

I'm thinking about how to makes this more efficient for this case. Even if it's not always the first node which is good for us, eg. it's the 5th node, we're still evaluating too many nodes. So, we have to have some sort of a threshold to make this improvement justifiable. I think the "first node fit" is always a case whose throughput is hard (or even impossible) to match with a parallel approach, simply due to the extra overhead. But I still suggest exploring different ways to make this more performant.

• You’re right: in this synthetic test the first node is always a hit, so batching/parallel evaluation adds overhead. We construct a batch (e.g., 24) and then validate 23 unnecessary candidates, which burns CPU and reduces throughput.
• Parallelism is unlikely to beat the “first-node-fit” path because of coordination and per-candidate predicate checks. In these cases, the optimal strategy is to use a single threaded approach.

In Short: match threads to your hit pattern. If the first candidate wins most of the time (≈90%), parallelism just adds overhead—run with a single thread and short-circuit on the first match.

Why and when parallelism helps:

• In our environment, nodes are split across pools with taints/tolerations. A node may meet resource requirements but still be unschedulable due to taints, so sequential scans waste time on many ineligible nodes.
• Evaluating multiple candidates concurrently reduces time-to-first-valid-node when viable nodes sit deeper in the list.

Concrete example (from the attached cluster snapshot):

• ~700 nodes total.
• ~310 nodes (~44%) idle at ~10% CPU/memory; the remainder sit at 50–70% utilization.
• Spark workloads are pinned to the more utilized pool via taints.
• With sequential evaluation, the first ~300 tryNodes are effectively dead ends; useful work begins only once we reach the tainted pool.
• Increasing parallelism (e.g., checking 10 nodes at once) cuts selection latency by roughly 10x because the checks are CPU-bound and don’t call the Kubernetes API.

Sophisticated future enhancement

• For synthetic or “first-node-fit” scenarios, prefer batch size = 1, single thread, and immediate exit on success.
• If hits typically occur deeper (e.g., around the 5th candidate or later), introduce a miss threshold before enabling parallel evaluation. Only pay the parallel overhead when early probes fail consistently.
• Consider an adaptive policy: start single-threaded; after N misses (tunable), switch to K-way parallel checks with cancellation on first success; decay back to single-threaded when top-of-list hits return.